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Abstract 

The paper contains a characterization of compact groups G C GL{V), 
where V is a finite dimensional real vector space, which have the following 
property SP: the family of convex hulls of G-orbits is a semigroup with 
respect to the Minkowski addition. If G is finite, then SP holds if and 
only if G is a Coxeter group; if G is connected then SP is true if and only 
if G is polar. In general, G satisfies SP if and only if it is polar and its 
Weyl group is a Coxeter group. 



1 Introduction 

A representation of a compact Lie group G in a finite dimensional Euclidean 
vector space O is called polar if there exists a linear subspace a C (which is 
said to be a Cartan subspace) such that 

(A) each orbit 0„ — Gv, where v € V, meets a; 

(B) for any w e a, the tangent space t„ = TuOu is orthogonal to a. 

It follows that the set Oi, n a is finite and a = t^l; for generic u £ a. Polar 
representations of compact Lie groups were defined and described in [4]. An 
example is the adjoint representation Ad of G in its Lie algebra g, with any 
Cartan subalgebra as a Cartan subspace. A more general example is the isotropy 
representation of a Riemannian symmetric space (an s-representation) . Let 
M = H/K be such a space, (} = t © be the Cartan decomposition, where f),? 
are Lie algebras of H,K, respectively, and the space D is Ad(i4r)-invariant. The 
representation Ad of if in U is polar since any maximal abelian subspace a of 
satisfies (A) and (B). By a result of Dadok ([4]), a representation of G is polar 
if and only if it is orbit equivalent to some s-representation: there exist H, K 
as above and an embedding G ^ K such that G is transitive on each K-ovh\i 
in 0. In the paper [4], the proof involves a case- by-case check. A conceptual 
proof was given in the papers [6], [7]. The paper [5] contains a classification 
of polar representations of complex reductive Lie groups (there is a natural 
extension of the definition onto this case) . The family of polar representations 
is not wide (see the papers [4]-[7] for detailed information). The survey [13] 
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contains a comparison of the property to be polar with other "good" properties 
of representations (for example, to have a free algebra of invariant polynomials). 

In this article, the polar representations of compact Lie groups are charac- 
terized by a semigroup property of their orbits. Let G be a compact subgroup 
of GL(t)); we say that G is polar if its identical representation is polar. For 
A,B Cti, 

A + B = {a + h: a e A, b G B} 

is the Minkowski sum of A and B. Let X denote the convex hull of a set X C o. 
Here is the semigroup property of G mentioned above: 

SP: the family {Ov}veo of convex hulls of orbits is a semigroup with respect 
to the Minkowski addition. 

The group 

W = {geG: ga=a}l (1) 

is said to be the Weyl group of G. Due to (B), it is finite. In the statement of 
the following theorem, which is the main result of the paper, Coxeter groups 
are treated as finite hnear groups generated by refiections in hyperplanes. 

Theorem 1. A compact linear group satisfies SP if and only if it is polar and 
its Weyl group is a Coxeter group. 

In the definition of a polar representation, it is not assumed that G is con- 
nected. However, this property depends only on the identity component of 
G; in particular, all finite linear groups are polar by definition. By the theorem, 
a finite hnear group satisfies SP if and only if it is a Coxeter group, including 
non-crystallographic ones. This is proved in Theorem 2, which also contains two 
other geometric criteria for SP (hence for a finite linear group to be Coxeter). 

Let G be connected. If G is polar, then generic orbits are isoparametric sub- 
manifolds in the ambient Euclidean space (a submanifold of the Euclidean space 
is called isoparametric if its normal bundle is flat and principal curvatures are 
constant for any parallel normal vector field). For isoparametric submanifolds 
of codimension greater than 2 the converse is true (i.e., they can be realized 
as principal orbits of polar groups, see [12]; in codimension 2, there are non- 
homogeneous examples). Am^ compact connected isoparametric submanifold is 
naturally associated with a Coxeter group (see, for example, [9, Section 6.3]). If 
the submanifold is an orbit of a polar group, then this Coxeter group coincides 
with the Weyl group. Thus, if G is connected, then SP holds if and only if G is 
polar. The proof of these facts uses the Morse theory; it would be interesting to 
know if there exists a direct elementary proof of SP for connected polar groups. 

Lot H be a subgroup of GL(ti) and £(t), H) be the family of all i?-invariant 
convex sets in o. Clearly, €{v,H) with the Minkowski addition is a semigroup. 
The following proposition is an essential step in the proof of the theorem. 

Proposition 1. If G is polar, then the mapping A ^ Ana, where a is a Cartan 
subspace, is a semigroup isomorphism between £(t), G) and C{a,W). 
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The semigroups of sets in topological groups were considered in papers [10], 
[11], [2], [1]. In M", one parameter semigroups are of the form {tQ}t>o, where 
Q is a convex set. In particular, the family of closed balls for any norm is a 
semigroup. For a left invariant Riemannian metric in a Lie group, the family of 
closed balls centered at the identity is also a semigroup of sets. This property 
holds for all left invariant inner metrics; moreover, it characterizes them (see [2]). 
Semigroups may be parameterized by more general objects than the numbers. In 
terms of [2], this defines a geometry on a group. The orbits of an s-representation 
are parameterized by points of a closed convex simplicial cone C (the Weyl 
chamber of the restricted root system). Thus, the semigroup of their convex 
hulls can be treated as a vector valued norm (in general, non-symmetric) on 
with values in C. 

The exposition is self-contained and elementary. Some of preliminary results 
were pubHshed in [8]. We refer to [3] and [13] for general facts on actions of 
groups. 

Throughout the paper, we keep the notation above. Furthermore, let / be 
a real function on a set X. The set of all x G X such that /(x) = maxy^x f{y) 
is called the peak set for f on X . A peak point is a point which is a peak set; 
in this case, we say that / has a peak on X. Int[(X) is the interior of a set 
X C [ in [ C 0. The space D is equipped with the inner product ( , ) such that 
G C 0(o). For v,u and compact X C t), set 

av = iy 0, 
K{u) = {v,u) , 
IJ.v{X) = Hx{v) = max^^x A„(a;), 
Py{X) = {xgX: X^{x) = fiv{X)}. 

Hx is the support function for X. Note that u is a critical point for A,, on 0„ 
if and only if u e o„, equivalently, if and only if u G a.v {gu -L v is the same as 
u _L Qv). Since Pv{Ou) is the peak set for A« on 0„, we get 

Pv{Ou) C a„. (2) 

For all compact X,Y C t and w € , we obviously have 

M,(X + y) = /x,(X)+M.(F), (3) 
P,{X + Y) = P,{X) + P,{Y). (4) 

The stable subgroup of i> S and its Lie algebra are denoted by G„, 
respectively. A point w e is said to be regular if G„ is minimal: Gu Q G„ 
impHes G„ = Gy, where m G 0. For X C o, X'^^^ is the set of all regular points 
in X; span(X) is the linear span of X. The algebra of Hnear operators ^ is 
denoted by L(ti), e is the unit of G, tt € L(d) is the orthogonal projection onto 
the Cartan subspace a, and = [0, cxd). 
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2 Preparatory material 



In this section, G is not assumed polar unless this is stated explicitly. Obviously, 
the sum of convex sets is convex and the sum of G-invariant sets is G- invariant. 
Hence the inclusion 

holds for all u, u e 0. The family {Ov}veV is a semigroup if and only if the 
equality holds for some u, v in every pair of orbits. 

The tangent space ty = TyOy may be identified with the quotient q/Qv or 
with the complementary subspace: 

iv = Qv^ Q/Qy = fl^ C g, 

where ± relates to some invariant inner product in g; then is ad(gt,)-invariant. 

Lemma 1. For any m G 0, there exists a neighborhood U of u in ttu such that 
u is a peak point on Ou for all Xy such that v G U. 

Proof. Clearly, m is a peak point for A„ on 0„ and (PXu is negative definite on 
iu- If V — u is small, then the latter is also true for cPXy', if v G a^, then u is 
a critical point for A,,. Hence has a strict local maximum on 0„, which is 
global if t; e is sufficiently close to u. □ 

Let G* be the closed convex cone hull of the shifted orbit O-y and Cy be 
the dual cone to it: 

Cy = {u e X) : {u,v - gv) >0 for all g€G}, (5) 
G„* = c1os(r+(v-6„)) . 

Clearly, v G Cy. It follows from (5) that 

UGCy <^ vGPuiOy). (6) 
Corollary 1. Cy is a closed convex generating cone in ay. 
Proof. Clearly, Cy is convex and closed. For all u,v G X) we have 

IJ-uiOy) = llyiOy), (7) 
UGCy <^ V G Cu, (8) 
UGPy{Ou) ^ VGPu{Oy). (9) 

Indeed, (7) is true since maxggc! {u,gv) = maxpgc {v,g~^u), (8) and (9) follow 
from (6), (7), and the evident equality Xu{v) = Xy{u). By (6), (9), and (2), 
Cy C Oy, moreover, Inta„(Gi,) ^ due to Lemma 1. □ 

According to Corollary 1, ty — Cy. Hence, 

G: = (G:na.) + t.. (10) 
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Note that C„ is pointed (and C* is generating) if and only if Int(O^) ^ 0. By 
(6) and (9), 

Pv{Ou)=C,nOu. (11) 
Corollary 2. Each G-orbit in meets Cy. 

Proof. Since G is compact, we have Xv{x) = HviOu) for some a; G 0„. □ 

Note that each orbit of any connected component of G also meets a« since 
there are critical points of on it; however, the intersection need not have a 
common point with Cy 

It follows from (11) that 

0„ n a = {v}. (12) 

Lemma 2. Let u & C^. If Xv has a peak on Ou at u, then G^ C Gu- 

Proof. If 5 G Gy, then gu G Cy by (8) since v = gv ^ gGu = Cgu. By (11), 
gu e Py{Ou). Since m is a peak point for A„, we get gu = u. □ 

Lemma 3. Let u G Oy and u ^ v. Set w = u — v. Then {Gu n Gy) ± w. 

Furthermore, w / ay and the set Cu nCu is contained in the hyperplane a,, Dw^ 
in ay, which is proper. The hyperplane separates C„ and Gy, moreover, u 
and V are strictly separated. 

Proof. If X G C„ n Gy, then Xx{u) = Xx{v) = iix{Oy) by (6). Hence {x,w) = 0. 
Further, w J- ay implies = \v\^ + \w\'^ > \v\'^ contradictory to m G Oy. For 
arbitrary a; G C„ and y G Gy, 

{w,x) = Xx{u) - Xx{v) > > Xy{u) - Xy{v) = {w,y) . 

The inequalities hold due to (6). If x ~ u or y = v, then at least one of the 
inequalities is strict; ii x = y & Gy Ci Cy, then we have equalities. □ 

Note that is the equidistant hyperplane for u and v. 

Corollary 3. The following assertions hold: 

(1) if u & Oy and v, then lnta^{Gy) fl C„ = 0; 

(2) if X € Inta„(C„), then C„ fl = GyX. 

Proof. The hyperplane strictly separates Inta„ {Gy) and C„ due to Lemma 3. 
Thus (1) is true. U g € G, x € Int^^ Gy, and gx & Gy, then 

X G Int„„ {Gy)nGg-iy. 

By (1), gv = V. Conversely, if gv = v, then gGy = Gy, hence gx G Gy. This 
proves (2). □ 

We omit the proof of the following lemma, which is standard. 
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Lemma 4. For any u € V^'^^ and its neighborhood V in 0, there exists a neigh- 
borhood U of u in X) with the following property: if v ^ U , then V nOy contains 
a unique critical point of Xu on O^, which is also a peak point for A„ on Oy. □ 

Lemma 5. A vector v G X) is regular if and only if Gy Q Gu for all u G ay. 

Proof. According to Lemma 2, for all u in the set U of Lemma 1 the reverse 
inclusion Gu C Gy holds; if v is regular, then Gu — Gy. Since U is open in ay, 
Gy C Gu for all u G ay. Conversely, let Gy C Gu for all u e ay. It follows from 
Corollary 2 that ajf^ ^0. Ifu € a^y^, then G„ — Gy since G„ is minimal. Thus, 
V is regular. □ 

We conclude this section with a proposition which combines the facts on 
polar groups that we need in the sequel. 

Proposition 2. Let G be polar, a be a Cartan subspace, and W be the Weyl 
group. Then a'^^^ is open and dense in a and a = tty for any v € a'^''^. Further- 
more, 

(i) if V G Ci^"^, g G G, and gv G a, then ga = a; 

(ii) if V G X)'^'^^, then Gy is a normal subgroup of finite index in the group 

G" = {gGG: gay = ay} = {gGG: gvG ay} 
and G^/Gy ^ W ; 

(iii) for all a G a, Wa = Oo fl a; 

(iv) for any a G a, irOa = Wa. 

Proof It follows from (A) that a""^^ ^ 0. Let v G a""^^. By (B), iy ± a. Hence 
a„ 3 0. If codimt„ > dim a, then dim(o/G') > dimo, contradictory to (A) and 
(B). Thus, tty = a. It is well known that 0'^°^ is open and dense in 0. Hence, 
o''*'^ is open in a; it follows from (A) and (B) that o''*'^ is dense in a. 

If w e a'^''^ and gv G 0, then gv G a"^**^. Hence a = agy. Since gay = ttgy and 
0,, = o, this implies (i). 

In (ii), we may assume v G a'^^^; then a„ = a. By Lemma 5, Gu = Gy for 
all u G a^'^^. Therefore, Ggy = Gy \{ g G G'" , Gy is normal in G", and we have 
G'"/Gy ^G''\a = W. By (i), n a = G^v; thus Wv = Oy^^ a. It follows from 
(B) that Oyf\a and W are finite. 

For a G a^^^, (iii) was proved above; for all a G a, (iii) is true since a''*'^ is 
dense in a. 

In (iv), the inclusion ttOq 3 Wa is obvious. By (iii), we have to prove that 
TrOa C n a. Otherwise, there exist u G Oa and h G a"^**^ such that 

\b{u) > max{Afc(x) : x G Oa D a}. 

Then Pb(Ou) n a = but this contradicts to (2) since a = at. □ 



6 



3 SP for finite finear groups 



If G is finite, then a„ = for all u € and 0'^°^ consists of u G such that 
Gv = {e}. The following lemma is a specification of Lemma 3 to this case. 

Lemma 6. For any w £ 0, the family of cones {Cg„}ggG defines the Dirichlet— 
Voronoi partition of for the orbit : 

Chv = {u€0: \u — hv\ = mm\u — ghv\}. (13) 

Proof. Let u G 0. By definition, —C* is the tangent cone to the convex polytope 
O,. at the vertex v. The dual cone C,, is uniquely determined by the following 
properties: it contains v and is bounded by hyperplanes which are orthogonal to 
extreme rays of C*. Let M.'^{v — gv), g € G,he such a ray. Then the equidistant 
hyperplane H for v and gv defines a face of Cv (note that \v\ = \gv\ impHes 
€ H). By (12), Oy nCv = {v}. Hence, m G C„ if and only if v is the nearest 
point for u in Oy. This proves (13) for h = e that is evidently sufficient. □ 

Corollary 4. If v G tJ^*^^ , then the action of G on the family of cones Cgy, 
g €i Q, is simply transitive. □ 

Theorem 2. Let G be finite. Then SP is equivalent to each of the following 
properties: 

(i) for any v G D"^®^, the functional Xy has a peak on each G-orbit in 0; 

(ii) G is a Coxeter group; 

(iii) if V & t)'"'*^, then C„ = Cy for any u from some neighborhood of v. 

Proof. SP =^ (i). Let v G 0™^^ u,w&\3, and + (5„ = O^. Set P = P^,(0„). 
Due to Corollary 2, we may assume u,w € Cy\ then m G P by (11). Clearly, 
Py{dy) = Py{Oy) = {v} aud Py^Oa) P. By (4), we have 

Pv{dy,) = v + P. 

Furthermore, v + P C. Int(Cu) since v G Int(Ci,), P Q Cy, and P is compact. 
Let E be the set of extreme points of v + P. Clearly, E C v + P C Int(C^). On 
the other hand, E C Oy, since extreme points of Py{Oyj) are extreme points of 
Oy,. By Corollary 3, Oy, CiCy = GyW. The stable subgroup Gy is trivial since 
V G V"""^. Therefore, v + P = {w} and P = {u}. 

(i) (ii). Let v G 0™^, u G Oy, u ^ v. Due to Lemma 6, we may assume 
that Cy and Cu have a common wall 

w = a n c„ 

which is contained in an equidistant hyperplane H such that IuXh W ^ 0. Let 
w G Int_f/(yV). Then u,v G Py,{Oy). It follows from (i) that w is not regular. 
Hence, Gy, ^ {e}. Let g G G«,\{e}. Then gCy ^ Cy by Corollary 4. Obviously, 
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w has a G^-invariant neighborhood U such that U C Cy U Cu\ this impHes 
gCy = Cu- Similar arguments shows that gCu = Cy. Therefore, g^Cy = Cy. 
By Corollary A, g"^ = e and the condition gCy = Cu uniquely determines g. The 
latter means that g is independent of the choice of w G IntH(yV). Hence g is 
identical on some open subset of H. Consequently, 5 is a nontrivial involution 
that fixes points of H . Thus, g is the reflection in H . Every pair of cones in 
the family {Cga}geG can be joined by a chain of these cones in such a way 
that consecutive ones have a common wall; since G acts freely on {Cgv}g^G by 
Corollary 4, G is generated by reflections. 

(ii) (iii), (ii) =^ SP. Let G be a Coxeter group and G be a Weyl 
chamber. Then G is a simplicial cone; let . . . ,Wn be a base in D such 
that G = X^fc=i R~'~^^fc and let Q:i,...,a„ be the dual base, which generates 
the dual cone G*. The group G^t, is generated by reflections in those walls of 
G that contain ruk (they correspond to aj with j / k). Hence, G^u^v C O,, ; 
moreover, G^^ u is a face of Oy which is orthogonal to Wk ■ This proves inclusions 
O^nG D (u-G*)nG and G^ 3 G. On the other hand, the set r\geG9i'" - ^*) 
is convex and contains v. Therefore, 

dy= f]giv-C*)=\J g{{v-C*)nC) (14) 
see seG 

and Cy = G. This proves (iii). Further, SP evidently holds for the families 
{{v — C*) n G}t,ec and {g{v — G*)}y^c for any g € G, hence for Oy. 

(iii) ^ (ii). Let Gt, = G„ = G for u in some open set U. Then G* = G* = 
G*, there G* is the dual cone to G. Since v G 0'^^^, we have Gy = {e}. Hence, 
we may assume that gu ^ hU ii g ^ h, taking a smaller U if necessary. This 
implies that each extreme ray of G* is of the form M"'"(u — gu), where g € G 
does not depend on u e U. Therefore, the linear operator 1 — 5 (where 1 is 
the identical transformation) maps U into some one dimensional subspace and 
is nontrivial. Hence it has rank 1. Since g is orthogonal, it is a reflection. It 
remains to note that the action of G on Oy is simply transitive and that every 
two vertices of a convex polytope can be joined by a chain of one dimensional 
edges. □ 

4 Proof of the main result 

For a polar G, a is a Cartan subspace, n is the orthogonal projection onto a, 
and W is the Weyl group; G^ denote the identity component of G. 

Lemma 7. Let G be polar. Then 

(1) for any G-invariant convex set Q ^ X), wQ = Q Da and Q = G{Q fl a); 

(2) for every W -invariant convex set A C a, GA is convex and wGA = A. 
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Proof. Clearly, nOv D D a ^ Ci a. Together with Proposition 2, (iii) and 
(iv), this implies ttO^ = 0„ fl a for all v G t. Since Q = [J^^q Oy, this proves 
the first equality in (1); the second follows from (A). 

If a G A, then Wa C A and we get ttGA = A since irOa = Wa for all a G a 
by Proposition 2, (iv). Thus, GA C Tr-^{A). Clearly, aDn-^iA) = A and the 
set TT~-^{A) is convex. For any g G G, the same is true for the set gA, the space 
ga, and the orthogonal projection Wg onto it. Hence GA = flgec '^g^id^)- This 
proves that GA is convex. Thus, (2) is true. □ 

Proof of Proposition 1. Clearly, tt induces a homomorphism G) £(o. W). 
Since G is polar, ttV = V n a for all F G £(d, G) by Lemma 7, (1). Hence the 
mapping a : V ^ V f\ a, V & £(t),G), which coincides with tt on £(o,G), 
is a homomorphism. It follows from Lemma 7, (1), that a is one-to-one. By 
Lemma 7, (2), GA G £(t),G) for any A G e:(a, VF) and ttGA = A. Hence a is 
surjective. □ 

Proof of Theorem 1. Let G be polar and be a Coxeter group. Then the 
family {VFajagQ is a semigroup due to Theorem 2. By Proposition 2, (iv), we 
have Oa = Wa for all a G a. It follows from Proposition 1 that {Ov}vex) is a 
semigroup, i.e., G satisfies SP. 

Conversely, let SP hold for^. Then, for each pair of G-orbits, there exist 
vectors u, v in them such that Ou + Oy = Ou+v ■ Clearly, 

=> iu + Ui ^Ui+v (15) 

Suppose u regular. We claim that 

a = a„ 

is a Cartan subspace. The condition (A) is obvious. By (15), dimt„ < dimt„+„; 
since u G a''®^, we have dimtu = dimtu_|_^. Therefore, iu = Ut+y = + ty. 
Moreover, ty = tu if t; is regular; then 

= 0^. (16) 

Thus, it is sufficient to prove that there exists a neighborhood U of u in o such 
that 

v€U =^ du + dy = du+y (17) 

to verify (B). Indeed, we may assume U C a''*'^. Then (17) implies (16) for all 
V G U; consequently, -L for all v € a. 
Let U be such that 

OynU = {v} (18) 

for each v G U. Since u G 0™^ is a peak point for A„ on 0„, the function A„ 
must have a peak on Oy for v near u by Lemma 4. The peak point v' G Oy 
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depends on v continuously in some neighborhood of u since u is regular and 
(PXu is nondegenerate on t^. Furthermore, v' G a = ttu since v' is a critical 
point for A„ on Oy. Thus, for sufficiently small U, (18) implies that v' = v if 
V & U. In other words, v in (18) is the peak point of A„ on Oy for all v E U. 
Further, we have Ou + Oy = Oyj for some w G 0. Clearly, A„ have a peak at u 
on Ou as well as on 0„, and the same is true for A„, v, 0„, and Oy. Therefore, 
Xu has a peak on 0„ + Oy at u + v. This implies that u + v is an extreme point 
for A^ on Ow but the set of extreme points of Ow coincides with since 
is homogeneous. Thus, u + v G Ow and we get (17). This proves the claim. 

Since G is polar, we may apply Proposition 1. According to it.^P for G 
impHes that the family {Oa n a}aea is a semigroup. The convex hull Wa is the 
least convex W^-invariant set which contains a, and the same is true for Oa and 
G. The mappin^Q QQaJceeps inclusions and is a bijcction by Proposition 1. 
Hence we have Oa H a = Wa for all a G a. Therefore, {VFajago is a semigroup; 
by Theorem 2, is a Coxeter group. □ 
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